Alarms for a system of smart media playback devices

ABSTRACT

One embodiment provides for a data processing system comprising a memory device to store instructions and one or more processors to execute the instructions stored on the memory device. The instructions cause the one or more processors to provide a virtual assistant to receive voice input at a media playback device associated with the data processing system. The virtual assistant, via the one or more processors, is configured to receive a voice command at the media playback device, where the voice command is to schedule an event, then determine a user account associated with the voice command, a type of command based on context information associated with the voice command, and a target device to which media playback associated with the event is to be scheduled.

CROSS-REFERENCE

This application is a divisional of co-pending U.S. application Ser. No.15/836,638, filed Dec. 8, 2017, which claims the benefit of U.S.Provisional Patent Application No. 62/514,651 filed Jun. 2, 2017, whichis hereby incorporated herein by reference.

BACKGROUND OF THE DISCLOSURE

Computing devices can be used to play and manage media items. Existingcomputing devices can play media that is stored on the local computingdevice, within a local media server or from a remote or cloud-basedmedia service. Media playback systems can be configured to allow a userto stream media from a computing device to one or more playback devices,such as a networked speaker, Bluetooth speaker, smart television,set-top-box, or other computer devices. However, during media playbackon a media playback system, incoming audio events may disrupt media thatis playing during the audio event.

SUMMARY OF THE DESCRIPTION

Embodiments described herein provide a media playback system thatenables a smooth transition between incoming audio events, such aspre-set alarms, and currently playing media.

One embodiment provides for a media playback device comprising a memorydevice to store instructions, one or more processors to execute theinstructions stored on the memory device, the instructions to cause theone or more processors to provide a playback queue manager to manage oneor more media playback queues including a set of media items associatedwith a scheduled event and a playback routing manager to determine anoutput destination for the set of media items based on contextassociated with the scheduled event, the playback routing manager toroute output of playback of the set of media items to one or more ofmultiple different connected media playback devices based on the contextassociated with the scheduled event.

One embodiment provides for a non-transitory, machine-readable mediumstoring instructions to cause one or more processors of an electronicdevice to perform operations including receiving a request to play aqueue of media item associated with a scheduled event; determiningwhether an existing playback queue is in effect on the electronicdevice; determining whether the queue of media items associated with thescheduled event is a transient queue; storing the existing playbackqueue as a secondary media queue in response to determining that theexisting playback queue is in effect and that the queue of media itemsassociated with the scheduled event is a transient queue; and playingthe queue of media items associated with the scheduled event until thescheduled event is dismissed.

One embodiment provides for a data processing system comprising a memorydevice to store instructions; one or more processors to execute theinstructions stored on the memory device, the instructions to cause theone or more processors to provide a virtual assistant to receive voiceinput at a media playback device associated with the data processingsystem, the virtual assistant, via the one or more processors, toreceive a voice command at the media playback device, the voice commandto schedule an event; determine a user account associated with the voicecommand; determine a type of command based on context informationassociated with the voice command; and determine a target device towhich media playback associated with the event is to be scheduled.

Other features of the present embodiments will be apparent from theaccompanying drawings and from the detailed description, which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example,and not limitation, in the figures of the accompanying drawings in whichreference numbers are indicative of origin figure, like references mayindicate similar elements, and in which:

FIG. 1 is a block diagram of an exemplary connected media playbacksystem, according to an embodiment;

FIG. 2 is a block diagram of a computing device for use within oneembodiment of a distributed media playback system;

FIG. 3 illustrates a connected group media playback system, according toone embodiment;

FIGS. 4A-4B illustrate zones that may be established within a connectedgroup media playback system, according to an embodiment;

FIGS. 5A-5D are flow diagrams illustrating operations associated withlogic to perform playback queue and routing management for a group ofsmart media playback devices, according to embodiments described herein;

FIG. 6 is a block diagram illustrating an exemplary API architecture,which may be used in some embodiments of the invention;

FIGS. 7A-7B are block diagrams of exemplary API software stacks,according to embodiments;

FIG. 8 is a block diagram of mobile device architecture, according to anembodiment;

FIG. 9 is a block diagram of one embodiment of a computing system; and

FIG. 10 is a block diagram of a virtual assistant, according to anembodiment.

DETAILED DESCRIPTION

Embodiments described herein provide a media playback system thatenables a smooth transition between incoming audio events, such aspre-set alarms, and currently playing media. Reference in thespecification to “one embodiment” or “an embodiment” means that afeature, structure, or characteristic described in conjunction with theembodiment can be included in at least one embodiment of the invention.The appearances of the phrase “in one embodiment” in various places inthe specification do not necessarily all refer to the same embodiment.

In the figures and description to follow, reference numbers areindicative of the figure in which the referenced element is introduced,such that an element having a reference number of N00 is firstintroduced in FIG. N. For example, an element having a reference numberbetween 100 and 199 is first shown in FIG. 1, while an element having areference number between 200 and 299 is first shown in FIG. 2, etc.Within a description of a given figure, previously introduced elementsmay or may not be referenced.

The processes and operations depicted in the figures that follow can beperformed via processing logic that includes hardware (e.g. circuitry,dedicated logic, etc.), software (as instructions on a non-transitorymachine-readable storage medium), or a combination of both hardware andsoftware. Although some of the processes are described below in terms ofsequential operations, it should be appreciated that some of theoperations described may be performed in a different order. Moreover,some operations may be performed in parallel rather than sequentially.Additionally, some operations may be indicated as optional and are notperformed by all embodiments.

Connected Media Playback System

FIG. 1 is a block diagram of an exemplary connected media playbacksystem 100, according to an embodiment. The connected media playbacksystem 100 can be configured to receive playback queues from a userdevice to one or more playback devices, such that the playback devicescan play and optionally manage playback of media items identified withinthe playback queue. In one embodiment, the connected media playbacksystem 100 can be configured to delegate media account credentialsassociated with a first user's device to a second user's device so thatthe second user's device can play back media items associated with thefirst user's media account. In some implementations, the connected mediaplayback system 100 can include one or more user devices, such as userdevice 102. The user device 102 can be a computing device such as asmartphone, laptop computer, tablet computer, smart watch, or any othercomputing device.

In some implementations, the user device 102 can include a mediaapplication 104 to manage media items (e.g., music tracks, playlists,albums, movies, television shows, radio talk shows, audio books,podcasts, etc.) on the user device 102. In some implementations, themedia application 104 can be a client application of a network mediaservice. For example, the media application 104 can be configured with auser's media service account credentials for accessing the media service142 on a media server 140. The user credentials can include, forexample, an account identifier (e.g., user name, phone number, deviceidentifier, account token, etc.) and a shared secret (e.g., password,encryption key, secret token, fingerprint data, etc.). The media service142 can provide cloud storage for the user's media items and can provideon demand access to media items, Internet radio, and/or other types ofmedia. A user can interact with various graphical user interfaces of themedia application 104 to access media items through a network 150 (e.g.,a local area network, wide area network, Wi-Fi network, cellularnetwork, the Internet, etc.). For example, the user can generate aplaylist of media items for playback, where the playlist may referencemedia items provided by the media service 142. When the user wishes toplay back the playlist, the media application 104 can load the playlistinto a playback queue 106 within memory of the user device 102. Themedia application 104 can then play the media items in playback queue106 using the display and/or speakers of the user device 102.

In some implementations, the user device 102 can stream a media item toa playback device. For example, while playing a media item in playbackqueue 106, media application 104 can send the audio and/or videoassociated with the playing media item to a playback device 130. Forexample, the playback device 130 can be a Wi-Fi speaker, a Bluetoothspeaker, a streaming media device (e.g., set-top-box), a smarttelevision, or any other device capable of receiving and/or presentingaudio and/or video data sent by user device 102. In this instance, themedia application 104 can play back media items received from the mediaservice 142, although the playback queue being played (e.g., playbackqueue 106) resides on the user device 102. The media application 104 cansend audio and/or video data to playback device 130 for presentationthrough the speakers and/or display of playback device 130. However, ifthe user device 102 is moved out of range of the playback device 130,such that the user device 102 loses the connection to playback device130, the playback device 130 will no longer receive the audio and/orvideo data. Accordingly, the playback device 130 may stop playing theaudio and/or video associated with the media items in playback queue106.

The connected media playback system 100 provided by embodimentsdescribed herein can additionally include one or more smart mediaplayback devices (e.g., smart playback device 120). The smart playbackdevice 120 can be a Wi-Fi speaker, a Bluetooth speaker, an in-carentertainment system, a streaming media device (e.g., set-top-box), asmart television, or any other device capable of receiving, managing,and playing a playback queue. Like the user device 102, the smartplayback device 120 can be configured with a processing system capableof executing applications, including a media application 122. The mediaapplication 122 within the smart playback device 120 can have the sameor similar capabilities and/or features as the media application 104 ofthe user device 102. For example, the media application 122 can includea playback queue 124 that includes information about media itemscurrently being played by the media application 122. In some instances,the playback queue 124 can identify media items provided by the mediaservice 142 on the media server 140. The smart playback device 120 canbe configured with a user's media account credentials for accessing themedia service 142 so that the media application 122 can obtain mediaitems identified in playback queue 124 from media service 142 throughthe network 150.

In some implementations, the user device 102 can send a playback queue106 to smart playback device 120. For example, the user device 102 canreceive a wireless signal 160 (e.g., Bluetooth, Wi-Fi, etc.) broadcastfrom the smart playback device 120. The smart playback device 120 canperiodically broadcast the wireless signal 160 so that other the userdevice 102 can find and connect to the smart playback device 120. Theuser device 102 can determine, based on the strength of the wirelesssignal 160, that the user device 102 is within a threshold distance of(e.g., proximate to) the smart playback device 120. In someimplementations, in response to determining that user device 102 isproximate to smart playback device 120, the user device 102 can presenta graphical user interface that a user can interact with to initiate atransfer of a media device playback queues between user device 102 andthe smart playback device 120. In some implementations, in response todetermining that user device 102 is proximate to smart playback device120 and that user device 102 is in a resting position (e.g., on a table,horizontal, not moving, etc.), the user device 102 can automaticallyinitiate a transfer of playback queue 106 to smart playback device 120.

In addition to the exchange of media playback queues, the user device102 and the smart playback device 120 can exchange other information,such as to-do lists, reminders, calendar events, and alarms. In oneembodiment, the smart playback device 120 can receive events from theuser device 102 or via an event server 110 that provides a calendarservice 112. The event server 110 can be any type of server, such as asingle physical server, a virtual server, or a cluster of physical orvirtual servers, including a cloud-based server. The calendar service112 can enable the synchronization of calendar events, tasks, reminders,and alarms between various devices connected via the network 150,including the user device 102 and the smart playback device 120. Themedia server 140 and the event server 110 can be part of a commoncloud-based server infrastructure. The calendar service 112 and themedia service 142 may be associated with the same account and can beaccessible using the same credentials.

The user device 102 and the smart playback device 120 can each executeapplications that communicate with the calendar service 112 on the eventserver 110. The user device 102 can execute a calendar application 107,or a similar application having an associated event list 108. The smartplayback device 120 can execute a calendar application 125 having anevent list 126 that may be synchronized with the event list 108 on theuser device 102 via the event server 110. Synchronization of the eventlists 108, 126 enable the user device 102 and the smart playback device120 to have a consistent view of the tasks, events, meetings, reminders,alarms, or other items for which the occurrence or impending occurrencethe user may wish to be notified. While a calendar application 107 isillustrated, events, meetings, reminders, alarms, or other alerts,alarms, and timers may be generated by or associated with variousapplications.

In addition to data exchange via media applications 104, 122 or calendarapplications 107, 125 the user device 102 and the smart playback device120 can each support a virtual assistant that enables voice control ofmedia playback and event scheduling. The user can direct the voiceassistant on either the user device 102 or the smart playback device 120to start or stop media playback or to schedule an event. The mediaplayback or scheduled event can be directed towards the user device 102or the smart playback device 120 using voice commands to the virtualassistant that is active in either device. The appropriate commands tofacilitate such operations may be relayed over the network 150 to theintended target device.

FIG. 2 is a block diagram of a computing device 200 for use within oneembodiment of a distributed media playback system. In one embodiment,the computing device 200 includes hardware and software that may besuitable for use within a user device or a smart playback device, suchas the user device 102 and/or the smart playback device 120 as inFIG. 1. The computing device 200 includes one or more speaker device(s)201 to enable media playback. Where the computing device 200 isimplemented as a smart playback device, the speaker device(s) 201 may beof higher quality relative to when the computing device is implementedas a user device.

The computing device 200 also includes a network interface 202 thatenables network communication functionality. The network interface 202can couple with one or more wireless radio(s) 203 to enable wirelesscommunication over one or more wireless networking technologies such as,but not limited to Wi-Fi and Bluetooth. In some implementations, thenetwork interface 202 may also support a wired network connection. Thecomputing device also includes a processing system 204 having multipleprocessor devices, as well as a system memory 210 which can be a systemvirtual memory having an address space that includes volatile andnon-volatile memory.

In one embodiment, the processing system 204 includes one or moreapplication processor(s) 205 to execute instructions for user and systemapplications that execute on the computing device. The processing systemcan also include a sensor processor to process and monitor a suite ofsensor devices 208 having sensors including, but not limited to motionsensors, light sensors, proximity sensors, biometric sensors, audiosensors (e.g., microphones), and image sensors (e.g., cameras). Thesensor processor 206 can enable low-power monitoring of always-onsensors within the suite of sensor devices 208. The sensor processor 206can allow the application processor(s) 205 to remain in a low powerstate when the computing device 200 is not in active use while allowingthe computing device 200 to remain accessible via voice or gesture inputto a virtual assistant 220. In one embodiment, the sensor processor 206or a similar low power processor within the processing system can enablelow power processing of media instructions provided by a media player225. The media player 225 may be a modular media player that is capableof playback of a variety of different audio and/or video media types,including but not limited to MPEG-2, MPEG-4, H.264, and H.265/HEVC. Inone embodiment, other formats may be supported via additional CODECplugins.

In one embodiment, the virtual assistant 220 is an intelligent automatedassistant system capable of engaging with a user in an integrated,conversational manner using natural language dialog. The virtualassistant 220 can invoke services external to the virtual assistant 220and, if necessary, the computing device 200, to obtain information orperform various actions requested by a user. The virtual assistant 220may be selectively given access to various software and hardwarecomponents within the computing device, including but not limited to thenetwork interface 202 to retrieve data via a network, media playbackapplications to initiate or stop playback of media files, or usercalendar data to schedule calendar events, tasks, reminders, or alarms.For example, and in one embodiment the virtual assistant 220 cancommunicate with an event scheduler 230 to schedule alarms or remindersvia a calendar system in communication with an event server, such as theevent server 110 of FIG. 1. Scheduled tasks can be added to an eventlist, such as events list 108 on a user device 102 or an event list 126within a smart playback device 120. The virtual assistant 220 can alsoinitiate playback of media. Where the computing device 200 is part of adistributed playback system, a distributed playback system module 212can perform operations to manage various aspects of media playback,including but not limited to a playback queue manager 214 to manage alist of media to be played via a distributed playback system and aplayback routing manager 216 to manage the specific elements of thedistributed playback system that are used to play media.

In one embodiment, the event scheduler 230 can exchange data with thedistributed playback system module 212. The data exchange can beperformed in response to input received via a user interface of thecomputing device 200 or a different computing device that participateswithin the distributed playback system. The data exchange can also beperformed in response to activity requested via the virtual assistant220. For example, and in one embodiment, an event scheduled via theevent scheduler 230 can be associated with a media playlist, such thatupon occurrence of the scheduled event, a playlist can be played via theplayback queue manager 214. For example, an alarm can be scheduled towake a user at a specific time. The alarm can be associated with aplaylist, such that one or more media elements will be played inassociation with or as a replacement for an alarm sound. In oneembodiment, a playlist can be associated with any event scheduled viathe event scheduler 230, including reminder or timer expiration events.

In one embodiment, the playback queue manager 214 can manage multiplesimultaneous playback queues, where the playback queues include one ormore past, present or future media elements to be played via thecomputing device 200. The playback queues can be loaded with individualmedia elements or playlists that specify multiple media elements. Theplayback queues can include locally stored media, media that will beretrieved for playback via a media server (e.g., media server 140 as inFIG. 1) or media that will be streamed from a local or remote mediastreaming server. Multiple types of media elements may be played overthe distributed playback system via the playback queue manager,including multimedia files such, as but not limited to music, musicvideos, and podcasts, including audio or video podcasts, or audio and/orvideo clips of current news, weather, or sports events.

Where one or more podcasts are selected for playback in response to theoccurrence of a scheduled event, podcast selection logic can select aspecific episode of a podcast for playback, such as the latest availableepisode of a podcast or the latest available unplayed podcast. Suchselection can be determined from explicit user preferences or based onlearned user preference information. The selection can also be performedbased on the age of the available unplayed podcasts relative to thecurrent date. In one embodiment, a podcast feed contains metadata thatindicates whether the podcast feed is associated with a serial podcastor a news-style podcast. Whether to play the earliest unplayed episodeor the latest episode can be determined at least in part based on suchmetadata.

In one embodiment, for example when setting a wake alarm, a news programor news channel may be selected for playback. The user may specific aspecific program or channel for playback in response to the wake alarm.Alternatively, the user can select a generic news category and logicassociated with the playback queue manager 214 can select the newsprogram or news channel to play based on selected user preferences. Inone embodiment, a news program preference can be determined based onnews topic preferences selected in a news program on a user device.

In one embodiment, when an existing playback queue is in effect during ascheduled event that causes playback of a different playback queue, theplayback queue manager 214 can manage the queues based on preferencesselected by a user. In one instance the playback queue manager 214 canbe configured to replace the existing playback queue with the newplayback queue, removing the queued items of the existing playback queueand replacing them with the items of the new playback queue. In suchinstance, the items selected for playback in response to the alarm orother scheduled event replace and subsume the previously queue items.Alternatively, the playback queue manager 214 can be configured toimplement a transient playback queue that is in effect only until thescheduled event is dismissed. After the scheduled event is dismissed,the playback items of the previously existing playback queue arerestored to the active queue and playback of the previously existingqueue can be resumed. The scheduled event can be dismissed via a voicecommand to the virtual assistant, via a user interface on the computingdevice 200, or a user interface of a user device connected to thecomputing device 200.

In one embodiment, the playback routing manager 216 can be used toselect a playback device within the distributed playback system to useto play a playback queue. Depending on the number of playback deviceswithin the distributed playback system, multiple different queues can beactive on multiple different playback devices or multiple differentplayback devices within the distributed playback system can be grouped.Grouped playback devices can share a common playback queue andsimultaneously play the same media. When a smart playback device isprovisioned, the playback device can be associated with one or moreusers and/or one or more user accounts. The smart playback device canalso be assigned a location and/or device type. In one embodiment,residential distributed playback network can be configured in whichmultiple user devices and play media via one or more smart playbackdevices within a residence. When a smart playback device is added to theresidential network, a room or location of each playback device can bespecified. An ownership can also be specified for each smart playbackdevice that indicates whether the smart playback device is associatedwith a single user or if the smart playback device is associated withmultiple users.

Groups and Zones Within a Distributed Connected Media Playback System

FIG. 3 illustrates a connected group media playback system 300,according to one embodiment. The connected group media playback system300 includes a plurality of connected devices 312, 316, 318 and one ormore connected speaker(s) (e.g., speaker 314). The connected devices312, 316, 318 can each be a variant of the user device 102 of FIG. 1.The speaker 314 can be a variant of the smart playback device 120 ofFIG. 1. Accordingly, the connected devices 312, 316, 318 and speaker 314may be any of the user devices or smart playback devices describedherein, including but not limited to a personal device such as ahandheld smart phone or other network enabled handheld device, apersonal computer or laptop, a wearable electronic device, a smart-TV,or a set-top-device. The speaker 314 can represent multiple speakerswithin a single location that are configured for stereo or surroundplayback or multiple speakers in different locations, where one or moreof the multiple speakers can be configured for multi-channel audioplayback. One or more speakers may also be configured with a display tofacilitate multimedia playback. In one embodiment, an in-carentertainment system as described can participate in the connected groupmedia playback system 300 and can include both a user device and smartplayback device.

The devices 312, 316, 318 and speakers 314 in the system may beconnected via a wide-area communications network, such as the Internet,or via local network connections such as Wi-Fi, Bluetooth, Ethernet, orother methods of connecting devices. Some devices and/or speaker(s) canalso connect via an ad hoc peer-to-peer network. For example, thedevices may be compliant with one or more wireless communicationprotocols. In some embodiments, the devices may also be adapted tosupport proprietary peer-to-peer protocols such as Apple® WirelessDirect Link (AWDL). Other similar protocols also may be adopted in orderto facilitate peer-to-peer communications. For peer-to-peer connections,the devices and speaker(s) may be connected in a full or partial meshconfiguration. For example, a device 312 may be connected to the speaker314 and an additional device 316 through a series of peer-to-peercommunication links. Collectively, the plurality of devices and thespeaker can form a playback group. Devices within a group cancommunicate and exchange playlists 322A-322D. In one embodiment theitems in the playlist can be loaded into a playback group via a playbackqueue manager (e.g., playback queue manager 214 of FIG. 2) associatedwith each device and speaker.

In an embodiment, multiple devices may be connected to and incommunication with a single control device but not with other deviceswithin the connected group media playback system 300. For example,device 316 can be configured as a control device and device 312, device318, and speaker 314 can each connect to device 316. In suchconfiguration, device 318 may not communicate with device 312 except viathe control device functionality provided by device 316. In variousembodiments, the control device for a group can be a user device or asmart media playback device.

In one embodiment, a device within the connected group media playbacksystem 300 can connect with other devices to download media resourceslisted within the playlists 322A-322D. In various embodiments, the itemsin each playlists 322-322D can explicitly specify a type and location inwhich the item is stored or logic within a distributed playback systemmodule (e.g., distributed playback system module 212 as in FIG. 2) canautomatically determine a location from which the media item may beretrieved. In one embodiment media items can be retrieved from storagelocal to the respective device, from a remote media server (e.g., mediaserver 140 of FIG. 1), or from other one of the connected devices basedon network connectivity and media availability.

In one embodiment, when devices 312, 316, 318 and speaker 314 areconnected within a group, media playback for the group can be directedvia a designated control device within the system. The control devicecan manage a single playlist for the group and the playlists that areused or displayed at the devices may be coordinated with the playlistmaintained by the control device. Group members can modify the groupplaylist and such modifications can be propagated to the group in areal-time manner. Editing and/or control options for a group member mayinclude pausing, rewinding, fast forwarding, skipping, adding, deleting,etc. For a single device playing the synchronized playlist,modifications to the playlist will be reflected in the playback of themultimedia. In one embodiment the specific type of modifications allowedto the playlist at the control device may be configured by the controldevice. Additionally, the control device can allow modifications to bemade by certain devices within the group while disallowing modificationsfor other devices in the group. Such configuration can be made at thedevice or account level. For example, the control device can configuredifferent user accounts associated with devices within the group to havedifferent levels of access, such that any device associated with adesignated account will have playlist modification access, while otherdevices associated with a different account will not have the ability tomodify group playlists. For example and in one embodiment, the varioususer accounts associated with the various devices can be part of afamily of related accounts that are associated with a family of users.In such embodiment, accounts designated as ‘parent’ accounts may havediffering levels of group playlist access relative to accountsdesignated as simply ‘adult’ accounts, while ‘child’ accounts can havedifferent levels of group playlist access relative to ‘parent’ or‘adult’ accounts.

In one embodiment, in addition to the speaker 314, device 312, 316, and318 can be or include a traditional device or a smart playback device.For example, device 312 can be a smart playback device instead of a userdevice. Alternatively, device 312 may be a user device that can functions a smart playback device. In one embodiment, device 312 can be awearable electronic device that can perform functions of a user device,and can at least temporarily serve as a smart playback device. Forexample, device 312 can be configured to play media associated with ascheduled event. For example, device 312 can be a smartwatch device oranother wearable electronic device. If, for example, via Bluetoothranging, the device 312 or a control device determines that the user isout of range or otherwise away from of any other playback device withinthe connected group media playback system 300, alarms or other eventreminders, and any associated playlists, can be played via the device312 instead of other playback devices in the group.

In one embodiment alarms and associated playlists can be played on allspeakers in the group that are local to and/or within a pre-determinedrange relative to a user for which the alarm or event is associated. Forexample, device 312 can be an in-car entertainment system that includesa traditional playback device and/or speaker 314 can be a smart playbackdevice associated with and controlled by device 312. If an alarm orevent were to occur while the user is in a vehicle containing the in-carentertainment system, the alarm or event, and any associated playlist,can play on the in-car entertainment system (e.g., device 312 and/orspeaker 314).

In one embodiment, a control device of the system can be aware of theoutput device state of each device within the group. For example, ifdevice 316 is a control device for a group, device 316 can be aware thatdevice 318 has multiple connected output device, such as an internalspeaker and a connected earbud 320. In such configuration, a playbackrouting manager (e.g., playback routing manager 216 of FIG. 2) can allowaudio routing logic internal to device 318 to determine the outputdevice with which to play any media or alerts or can explicitly directthe device 318 to output media audio via a specific output deviceattached to the device 318.

In one embodiment devices within the connected group media playbacksystem 300 can be sub-grouped into zones. For example, where multiplesinstances of the speaker 314 and/or the devices 312, 316, 318 arestationary or semi-stationary devices, a precise or general location canbe associated with the speaker or device. For example, device 312 can bea stationary or semi-stationary desktop computing device and the speaker314 can represent one or more stationary or semi-stationary smart mediaplayback device. In this example, a semi-stationary device or speaker isa device or speaker that is intended to remain in a single location butcan be moved to a different location, while a stationary device orspeaker is designed to be fixed in place, such as a device or speakerassociated with a wall mounted smart thermostat, wall mounted computingdevice, wall or ceiling mounted smart-speaker, or a smart appliance suchas a smart kitchen appliance.

In various embodiments, multiple zone configuration techniques can beused. In one embodiment, a zone can be a region or group of locationsassociated with each device. The associated zone can be used todetermine playback and routing for a playback queue or playlist. In oneembodiment, a location or identifier can be assigned to devices and,based on the location or identifier, the device can be automaticallyassociated with a zone. Alternatively, a device can be explicitlyassigned both a zone and an identifier, where the zone can be used toindicate a general location of the device and the identifier canindicate a user or users associated with the device and/or whether thedevice is identified as a personal device for one or more closelyconnected or similar users or whether the device is a common or communaldevice that is shared by multiple users that may have dissimilarconfiguration or media preferences. In the case of a communal device(including a communal smart speaker device), media preferences learnedabout a user will not be used to influence media selection whenautomated media playback is requested. Additionally, media requested forplayback will not influence the media preferences of users associatedwith the device. In contrast, a personal device can inherit andinfluence the learned media preferences associated with a user.

FIGS. 4A-4B illustrate zones that may be established within a connectedgroup media playback system, according to an embodiment. FIG. 4Aillustrates a first set of zones. FIG. 4B illustrates a second set ofzones. In one embodiment, each smart playback device within theconnected group can be as assigned an identifier and associated with oneor more user devices. The devices can also be assigned to a zone or thedevices can be assigned a location that can be correlated with a zone.One or more control devices in the network can determine which zone orwhich device within a zone will be assigned playback of an alert task ormedia queue. The zones illustrated are exemplary of one embodiment andare not limiting as to all embodiments. Various embodiments may allowthe definition or determination of a variety of different zones. In oneembodiment, a pre-defined set of zones are provided and additional zonesmay be user defined.

FIG. 4A illustrates a first set of zones including a utility zone 400, akitchen zone, and a dining room zone 420. The utility zone 400 caninclude multiple smart media playback devices, including a first device404 assigned to a laundry room location, and a second device 406assigned to a garage location. The utility zone 400 can be a pre-definedzone or a user-defined zone to which the devices of the illustratedlocations can associated. Although not illustrated, an additional devicecan be a mobile smart media playback device located within a vehicle orcan be an in-car information and/or entertainment (e.g., infotainment)system installed within a vehicle. The kitchen zone 410 can include afourth smart media playback device 414, such as a wall-mounted smartmedia playback device, in a first kitchen location. The kitchen zone 410can also include a smart appliance device 416, such as a smartrefrigerator device, within a second kitchen location. The dining roomzone 420 can include multiple co-located devices 422A, 422B that can begrouped as a single virtual device configured to play back differentaudio channels.

FIG. 4B illustrates a second set of zones including a living room zone430 and a bedroom zone. The living room zone 430 includes a fifth smartplayback device 436 having a living room location. The bedroom zone 440can include a sixth smart media playback device 442 having a firstbedroom location and a seventh smart media playback device 444 having asecond bedroom location.

In one embodiment, all the smart playback devices within each zoneillustrated within FIG. 4A-4B can be assigned to a residential group.All devices within the residential group can be synchronized tosimultaneously play the same media, alerts, alarms, notifications,and/or playlist associated with an alert or alarm. In one embodimentdevices within each zone can be synchronized to play media or alerts asa group. In one embodiment, for certain types of alerts, one or moredevices within a zone can temporarily decouple from other devices in thezone and initiate playback of a playlist in response to an alert oralarm without changing the behavior of other playback devices within thezone.

For example, the sixth smart media playback device 442 at the firstbedroom location can be associated with a different user than theseventh smart media playback device 444 at the second bedroom location.Each user can configure different morning alarms to play at differenttimes, with each alarm associated with separate media playlists. Eventhrough the sixth smart media playback device 442 and the seventh smartmedia playback device 444 are within the same zone (e.g., bedroom zone440), at the occurrence of specific alarms associated with specificusers, the respective devices can temporarily decouple from the zone toplay a user specific alert or alarm and/or a playlist associated withthe user specific alert or alarm.

One or more of the smart media playback devices within the group can becapable of audio output and voice command input, but may lack displaydevices on which scheduled events (e.g., alerts, alarms, timers,reminders, etc.) can be displayed. To give proper context to a scheduledevent, each scheduled event can be assigned a label. The label can bespoken by the media playback device upon occurrence of the event. In oneembodiment, each scheduled alert, alarm, or reminder can beautomatically associated with a label based on voice input provided tothe virtual assistant when scheduling the event. For example a user canissue a voice command to a virtual assistant at a smart media playbackdevice to scheduler a reminder to perform a future event (e.g., “Remindme to take the BBQ off the grill in 10 minutes.”). The virtual assistantcan then request confirmation of the event (e.g., “OK, I will remind youto take the BBQ off the grill in 10 minutes.”) A timer event can then bescheduled to count down for 10 minutes or an event can be scheduled tooccur 10 minutes into the future. At the occurrence of the event or theexpiration of the timer, the voice assistant can issue a voice alert tothe user (e.g., “Time to take the BBQ off the grill!”). The voice alertcan be accompanied with a pre-selected sound. In addition to issuing thealert, the smart media playback device can also begin playback of apre-selected set of media items, such as a media playlist.

Context Based Scheduling of Alerts

In one embodiment, voice commands issued to a virtual assistant on asmart media playback device can cause alerts to be scheduled in acontext sensitive manner. A user can issue a voice command to any smartmedia playback device within a group of smart media playback devices andthe command can be executed by or scheduled to the appropriate smartmedia playback device based on context. In various embodiments, andbased on context information associated with an alert on alarm, therouting logic can either schedule the alert or alarm to occur on aspecific device, route audio output associated with the alert or alarmto the specific device, or direct the specific device to play the alertor alarm via media playback logic on the specific device.

For example, wakeup alarms can be intelligently scheduled to the smartmedia playback device within a user's bedroom, even if the user were toissue a voice command to schedule a wakeup on a different smart mediaplayback device. For example, a user can use a voice assistant toschedule the wakeup event on the fourth smart media playback device 414within the kitchen zone 410. Routing logic within the fourth smart mediaplayback device 414 can then intelligently schedule the event to occuron the sixth smart media playback device 442 or the seventh smart mediaplayback device 444 in the bedroom zone. In one embodiment, a shoppingrelated reminder can be intelligently scheduled to a smart mediaplayback device associated with a smart appliance. For example, areminder to buy grocery items can be verbally requested via the sixthsmart media playback device 442 in the bedroom zone 440 and the remindercan be intelligently routed to a smart appliance device 416, such as asmart refrigerator device.

In one embodiment, events can be scheduled to multiple media playbackdevices based on the context of the request. For example, a user canissue a voice command to the virtual assistant (e.g., “Wake up the kidsat 8:00 AM tomorrow.”). Based on account or configuration settingsassociated with the group of smart media playback devices, the virtualassistant can determine the users identified by “the kids” and schedulea wakeup event to one or more smart media devices associated with theidentified users. The ability to schedule events for different users canbe limited based on settings associated with the users or an accounttype associated with the users. In one embodiment, the user issuing thevoice command can be determined based on the device that receives thecommand. In one embodiment, the voice assistant is able to determine theuser that issues the command via voice analysis and comparison of therequesting voice to a voice pattern associated with a user.

FIGS. 5A-5D are flow diagrams illustrating operations associated withlogic 500 to perform playback queue and routing management for a groupof smart media playback devices, according to embodiments describedherein. FIG. 5A illustrates operations to manage playback queues on asmart media playback device in response to a playback event. FIG. 5Billustrates operations to perform context based scheduling of alerts.FIG. 5C illustrates operations for media playback at a subset of deviceswithin a zone. FIG. 5D illustrates notification options for eventlabels. The operations can be implemented by a smart media playbackdevice including a virtual assistant, as described herein.

As shown in FIG. 5A, the logic 500 can receive a request to play a queueof media items associated with a scheduled event, as shown at block 502.The request can be to playback media to a single smart media playbackdevice, to a zone of devices, or to an entire group of devices. Thelogic 500 can then determine at block 503 whether an existing playbackqueue is in effect. If no existing playback queue is in effect (e.g., nomedia items are playing on the smart media devices), the logic 500 canset the received queue of media items as the current playback queue forthe smart media device, zone, or group of smart media devices, as shownat block 504. If an existing playback queue is in effect at block 503,the logic 500 can determine at block 505 whether the received queue ismarked as a transient queue. In one embodiment, a transient queue is aplayback queue that is in effect only until the scheduled event isdismissed. For example, a playback queue associated with an alarm eventcan be a transient queue. The transient status of a playback queue canbe determined based on the type of event associated with the playbackqueue or can be determined via metadata for the playback queue. If thereceived queue is a transient queue, the logic can proceed to block 504to set the received queue of media items as the current queue. If, atblock 505, the logic determines that the received queue is not atransient queue, the logic 500 can store the current queue of mediaitems as a secondary queue, as shown at block 506, before proceeding toblock 504.

The received queue of media items associate with the scheduled event canbe in effect until the logic 500 receives dismissal of the scheduledevent at block 508. Dismissal of the scheduled event can includecancelling an alert or notice associated with the triggering of theevent. The dismissal can occur via a voice command to a virtualassistant, via a user interface on the device issuing the alert, or viauser interface of a device connected to the device issuing the alert.After dismissal of a scheduled event, the logic 500 can determine, atblock 509, whether a secondary queue exists that should be restored. Ifa secondary queue exists that represents the active queue at the time inwhich the event was received, the logic 500 can restore the secondaryqueue of media items as the current queue, as shown at block 510.Otherwise, the logic 500 can end or pause playback of the current queue,as shown at block 512, upon dismissal of the scheduled event.

As shown in FIG. 5B, the logic 500 can receive a voice command at afirst smart media device to schedule an event, as shown at block 514.After receiving the command, the logic 500 can perform an operation 516to determine a user (e.g., user account) associated with the voicecommand. The user or user account associated with a voice command can bedetermined based on a user account associated with the smart mediaplayback device from which the voice command originated or via voiceanalysis of the voice command. The logic 500 can then perform anoperation 518 to determine a type of command based on contextinformation within the voice command. For example, the voice commandreceived at block 514 can explicitly state, “remind me,” to “do a task”at “a specified time,” and natural language processing logic candetermine that the type of command is to set a reminder. The voicecommand can also state, for example, “wake me up” at “a specified time”(e.g., 8:00 AM). The term, “wake me up” can indicate that the voicecommand is to set a wake alarm.

Having determined the type of command, the logic 500 can perform anoperation 520 to determine a target device to which the media playbackassociated with the event is to be scheduled. For example, the logic 500can determine that a wake alarm is to be scheduled for playback at asmart media playback device within a bedroom location of the identifieduser. The logic 500 can the perform an operation to output 522 a verbalrequest for confirmation of the determined type of command and targetmedia playback device. For example, the logic 500 can output a verbalstatement for confirmation that states “OK, I will wake you up at yourbedroom device at 8:00 AM,” in response to a request that does notexplicitly state a target device for the alert. The user can verballyacknowledge that the command was correctly stated or can issue acorrection. Based on an acknowledgement or correction, as determined atblock 523, the logic 500 can confirm whether the determined type andtarget are correct. Where a correction is issued, the logic 500 canreturn to operation 520 to re-determine the target device. If thecommand was correctly stated, the logic 500 can schedule the event andassociated media playback to the determined target media playbackdevice, as shown at block 524.

As shown in FIG. 5C, logic 500 can receive a request for media playbackwithin a zone in response to a scheduled event, as shown at block 526.The request can include a playlist of media items for playback. Thelogic 500 can perform an operation 527 to determine if an existingplayback queue is active for the zone. If no existing playback queue isactive for the zone, the logic 500 can perform an operation 528 to loadmedia items in a playlist into a media playback queue for the zone,followed by an operation 529 to play the media playback queue for thezone.

If operation 527 determines that an existing media playback queue isactive for the zone, the logic 500 can select a media playback devicewithin the zone and temporary detach the media playback device from thezone. The media playback device selected can vary based on the scheduledevent. If the scheduled event is an alert of an alarm associated with aspecific user, the selected media playback device can be a smart mediaplayback device in the room of the user associated with the alert or aspecific smart media playback device of the user associated with thealert. In one embodiment, the media playback device may be a userdevice, such as a wearable electronic device.

Once a media playback device is selected, the logic 500 can perform anoperation 532 to load media items on the received playlist into themedia playback queue of the detached media playback device and performan additional operation 534 to play the loaded media playback queue atthe detached device. The logic 500 can then restore the detached mediaplayback device to the zone upon dismissal of the scheduled event, asshown at block 536.

The media playback device selected to play an alarm or notification, ora playlist associated with an alarm or notification may or may not havean internal display device, and some implementations of the smart mediaplayback device do not have displays to display labels associated withevents. Accordingly, the notification mechanism for a label can varybased on device type. As shown in FIG. 5D, in response to receivingnotification of a scheduled event at a smart media playback deice atblock 540, the logic 500 can determine whether the smart media playbackdevice has a display screen suitable for displaying notifications, asshown at block 541. If the smart media playback device has such display,the logic 500 can display a notification or label in conjunction with analert associated with the scheduled event, as shown at block 542. If thesmart media playback device does not have a display capable ofdisplaying labels or notifications, the logic 500 can verbally announcea label of the scheduled event in conjunction with the alert, as shownat block 544. The label can be determined using a text label entered bya user or via a natural language processor if the event is scheduled viaa virtual assistant. It will be understood that, in some embodiments, asmall or low resolutions display may be present on a media playbackdevice or smart media playback device that may be able to convey limitedinformation to a user without being of sufficient size or resolution todisplay labels or notifications. Such devices may be classified aslacking a display and can be configured to verbally announce the labelsassociated with scheduled events.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information data will comply withwell-established privacy policies and/or privacy practices. Inparticular, such entities should implement and consistently use privacypolicies and practices that are generally recognized as meeting orexceeding industry or governmental requirements for maintaining personalinformation data private and secure. For example, personal informationfrom users should be collected for legitimate and reasonable uses of theentity and not shared or sold outside of those legitimate uses. Further,such collection should occur only after receiving the informed consentof the users. Additionally, such entities would take any needed stepsfor safeguarding and securing access to such personal information dataand ensuring that others with access to the personal information dataadhere to their privacy policies and procedures. Further, such entitiescan subject themselves to evaluation by third parties to certify theiradherence to widely accepted privacy policies and practices.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, in the caseof advertisement delivery services, the present technology can beconfigured to allow users to select to “opt in” or “opt out” ofparticipation in the collection of personal information data duringregistration for services. In another example, users can select not toprovide location information for targeted content delivery services. Inyet another example, users can select to not provide precise locationinformation, but permit the transfer of location zone information.

Embodiments described herein include one or more application programminginterfaces (APIs) in an environment in which calling program codeinteracts with other program code that is called through one or moreprogramming interfaces. Various function calls, messages, or other typesof invocations, which further may include various kinds of parameters,can be transferred via the APIs between the calling program and the codebeing called. In addition, an API may provide the calling program codethe ability to use data types or classes defined in the API andimplemented in the called program code.

An API allows a developer of an API-calling component (which may be athird-party developer) to leverage specified features provided by anAPI-implementing component. There may be one API-calling component orthere may be more than one such component. An API can be a source codeinterface that a computer system or program library provides in order tosupport requests for services from an application. An operating system(OS) can have multiple APIs to allow applications running on the OS tocall one or more of those APIs, and a service (such as a programlibrary) can have multiple APIs to allow an application that uses theservice to call one or more of those APIs. An API can be specified interms of a programming language that can be interpreted or compiled whenan application is built.

In some embodiments, the API-implementing component may provide morethan one API, each providing a different view of or with differentaspects that access different aspects of the functionality implementedby the API-implementing component. For example, one API of anAPI-implementing component can provide a first set of functions and canbe exposed to third party developers, and another API of theAPI-implementing component can be hidden (not exposed) and provide asubset of the first set of functions and also provide another set offunctions, such as testing or debugging functions which are not in thefirst set of functions. In other embodiments, the API-implementingcomponent may itself call one or more other components via an underlyingAPI and thus be both an API-calling component and an API-implementingcomponent.

An API defines the language and parameters that API-calling componentsuse when accessing and using specified features of the API-implementingcomponent. For example, an API-calling component accesses the specifiedfeatures of the API-implementing component through one or more API callsor invocations (embodied for example by function or method calls)exposed by the API and passes data and control information usingparameters via the API calls or invocations. The API-implementingcomponent may return a value through the API in response to an API callfrom an API-calling component. While the API defines the syntax andresult of an API call (e.g., how to invoke the API call and what the APIcall does), the API may not reveal how the API call accomplishes thefunction specified by the API call. Various API calls are transferredvia the one or more application programming interfaces between thecalling (API-calling component) and an API-implementing component.Transferring the API calls may include issuing, initiating, invoking,calling, receiving, returning, or responding to the function calls ormessages; in other words, transferring can describe actions by either ofthe API-calling component or the API-implementing component. Thefunction calls or other invocations of the API may send or receive oneor more parameters through a parameter list or other structure. Aparameter can be a constant, key, data structure, object, object class,variable, data type, pointer, array, list or a pointer to a function ormethod or another way to reference a data or other item to be passed viathe API.

Furthermore, data types or classes may be provided by the API andimplemented by the API-implementing component. Thus, the API-callingcomponent may declare variables, use pointers to, use or instantiateconstant values of such types or classes by using definitions providedin the API.

Generally, an API can be used to access a service or data provided bythe API-implementing component or to initiate performance of anoperation or computation provided by the API-implementing component. Byway of example, the API-implementing component and the API-callingcomponent may each be any one of an operating system, a library, adevice driver, an API, an application program, or other module (itshould be understood that the API-implementing component and theAPI-calling component may be the same or different type of module fromeach other). API-implementing components may in some cases be embodiedat least in part in firmware, microcode, or other hardware logic. Insome embodiments, an API may allow a client program to use the servicesprovided by a Software Development Kit (SDK) library. In otherembodiments, an application or other client program may use an APIprovided by an Application Framework. In these embodiments, theapplication or client program may incorporate calls to functions ormethods provided by the SDK and provided by the API or use data types orobjects defined in the SDK and provided by the API. An ApplicationFramework may in these embodiments provide a main event loop for aprogram that responds to various events defined by the Framework. TheAPI allows the application to specify the events and the responses tothe events using the Application Framework. In some implementations, anAPI call can report to an application the capabilities or state of ahardware device, including those related to aspects such as inputcapabilities and state, output capabilities and state, processingcapability, power state, storage capacity and state, communicationscapability, etc., and the API may be implemented in part by firmware,microcode, or other low level logic that executes in part on thehardware component.

The API-calling component may be a local component (i.e., on the samedata processing system as the API-implementing component) or a remotecomponent (i.e., on a different data processing system from theAPI-implementing component) that communicates with the API-implementingcomponent through the API over a network. It should be understood thatan API-implementing component may also act as an API-calling component(i.e., it may make API calls to an API exposed by a differentAPI-implementing component) and an API-calling component may also act asan API-implementing component by implementing an API that is exposed toa different API-calling component.

The API may allow multiple API-calling components written in differentprogramming languages to communicate with the API-implementing component(thus the API may include features for translating calls and returnsbetween the API-implementing component and the API-calling component);however, the API may be implemented in terms of a specific programminglanguage. An API-calling component can, in one embedment, call APIs fromdifferent providers such as a set of APIs from an OS provider andanother set of APIs from a plug-in provider and another set of APIs fromanother provider (e.g. the provider of a software library) or creator ofthe another set of APIs.

FIG. 6 is a block diagram illustrating an exemplary API architecture,which may be used in some embodiments of the invention. As shown in FIG.6, the API architecture 600 includes the API-implementing component 610(e.g., an operating system, a library, a device driver, an API, anapplication program, software or other module) that implements the API620. The API 620 specifies one or more functions, methods, classes,objects, protocols, data structures, formats and/or other features ofthe API-implementing component that may be used by the API-callingcomponent 630. The API 620 can specify at least one calling conventionthat specifies how a function in the API-implementing component receivesparameters from the API-calling component and how the function returns aresult to the API-calling component. The API-calling component 630(e.g., an operating system, a library, a device driver, an API, anapplication program, software or other module), makes API calls throughthe API 620 to access and use the features of the API-implementingcomponent 610 that are specified by the API 620. The API-implementingcomponent 610 may return a value through the API 620 to the API-callingcomponent 630 in response to an API call.

It will be appreciated that the API-implementing component 610 mayinclude additional functions, methods, classes, data structures, and/orother features that are not specified through the API 620 and are notavailable to the API-calling component 630. It should be understood thatthe API-calling component 630 may be on the same system as theAPI-implementing component 610 or may be located remotely and accessesthe API-implementing component 610 using the API 620 over a network.While FIG. 6 illustrates a single API-calling component 630 interactingwith the API 620, it should be understood that other API-callingcomponents, which may be written in different languages (or the samelanguage) than the API-calling component 630, may use the API 620.

The API-implementing component 610, the API 620, and the API-callingcomponent 630 may be stored in a machine-readable medium, which includesany mechanism for storing information in a form readable by a machine(e.g., a computer or other data processing system). For example, amachine-readable medium includes magnetic disks, optical disks, randomaccess memory; read only memory, flash memory devices, etc.

FIGS. 7A-7B are block diagrams of exemplary software stacks 700, 710,according to embodiments. FIG. 7A shows an exemplary API software stack700 in which applications 702 can make calls to Service A or Service Busing Service API and to Operating System 704 using an OS API.Additionally, Service A and Service B can make calls to Operating System704 using several OS APIs.

FIG. 7B shows an exemplary software stack 710 including Application 1,Application 2, Service 1, Service 2, and Operating System 704. Asillustrated, Service 2 has two APIs, one of which (Service 2 API 1)receives calls from and returns values to Application 1 and the other(Service 2 API 2) receives calls from and returns values to Application2. Service 1 (which can be, for example, a software library) makes callsto and receives returned values from OS API 1, and Service 2 (which canbe, for example, a software library) makes calls to and receivesreturned values from both OS API 1 and OS API 2. Application 2 makescalls to and receives returned values from OS API 2.

Additional Exemplary Computing Devices

FIG. 8 is a block diagram of a device architecture 800 for a mobile orembedded device, according to an embodiment. The device architecture 800includes a memory interface 802, a processing system 804 including oneor more data processors, image processors and/or graphics processingunits, and a peripherals interface 806. The various components can becoupled by one or more communication buses or signal lines. The variouscomponents can be separate logical components or devices or can beintegrated in one or more integrated circuits, such as in a system on achip integrated circuit.

The memory interface 802 can be coupled to memory 850, which can includehigh-speed random-access memory such as static random access memory(SRAM) or dynamic random access memory (DRAM) and/or non-volatilememory, such as but not limited to flash memory (e.g., NAND flash, NORflash, etc.).

Sensors, devices, and subsystems can be coupled to the peripheralsinterface 806 to facilitate multiple functionalities. For example, amotion sensor 810, a light sensor 812, and a proximity sensor 814 can becoupled to the peripherals interface 806 to facilitate the mobile devicefunctionality. One or more biometric sensor(s) 815 may also be present,such as a fingerprint scanner for fingerprint recognition or an imagesensor for facial recognition. Other sensors 816 can also be connectedto the peripherals interface 806, such as a positioning system (e.g.,GPS receiver), a temperature sensor, or other sensing device, tofacilitate related functionalities. A camera subsystem 820 and anoptical sensor 822, e.g., a charged coupled device (CCD) or acomplementary metal-oxide semiconductor (CMOS) optical sensor, can beutilized to facilitate camera functions, such as recording photographsand video clips.

Communication functions can be facilitated through one or more wirelesscommunication subsystems 824, which can include radio frequencyreceivers and transmitters and/or optical (e.g., infrared) receivers andtransmitters. The specific design and implementation of the wirelesscommunication subsystems 824 can depend on the communication network(s)over which a mobile device is intended to operate. For example, a mobiledevice including the illustrated device architecture 800 can includewireless communication subsystems 824 designed to operate over a GSMnetwork, a CDMA network, an LTE network, a Wi-Fi network, a Bluetoothnetwork, or any other wireless network. In particular, the wirelesscommunication subsystems 824 can provide a communications mechanism overwhich a media playback application can retrieve resources from a remotemedia server or scheduled events from a remote calendar or event server.

An audio subsystem 826 can be coupled to a speaker 828 and a microphone830 to facilitate voice-enabled functions, such as voice recognition,voice replication, digital recording, and telephony functions. In smartmedia devices described herein, the audio subsystem 826 can be a highquality audio system including support for virtual surround sound.

The I/O subsystem 840 can include a touch screen controller 842 and/orother input controller(s) 845. For computing devices including a displaydevice, the touch screen controller 842 can be coupled to a touchsensitive display system 846 (e.g., touch-screen). The touch sensitivedisplay system 846 and touch screen controller 842 can, for example,detect contact and movement and/or pressure using any of a plurality oftouch and pressure sensing technologies, including but not limited tocapacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with a touch sensitive displaysystem 846. Display output for the touch sensitive display system 846can be generated by a display controller 843. In one embodiment, thedisplay controller 843 can provide frame data to the touch sensitivedisplay system 846 at a variable frame rate.

In one embodiment, a sensor controller 844 is included to monitor,control, and/or processes data received from one or more of the motionsensor 810, light sensor 812, proximity sensor 814, or other sensors816. The sensor controller 844 can include logic to interpret sensordata to determine the occurrence of one of more motion events oractivities by analysis of the sensor data from the sensors.

In one embodiment, the I/O subsystem 840 includes other inputcontroller(s) 845 that can be coupled to other input/control devices848, such as one or more buttons, rocker switches, thumb-wheel, infraredport, USB port, and/or a pointer device such as a stylus, or controldevices such as an up/down button for volume control of the speaker 828and/or the microphone 830.

In one embodiment, the memory 850 coupled to the memory interface 802can store instructions for an operating system 852, including portableoperating system interface (POSIX) compliant and non-compliant operatingsystem or an embedded operating system. The operating system 852 mayinclude instructions for handling basic system services and forperforming hardware dependent tasks. In some implementations, theoperating system 852 can be a kernel.

The memory 850 can also store communication instructions 854 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers, for example, to retrieve webresources from remote web servers. The memory 850 can also include userinterface instructions 856, including graphical user interfaceinstructions to facilitate graphic user interface processing.

Additionally, the memory 850 can store sensor processing instructions858 to facilitate sensor-related processing and functions; telephonyinstructions 860 to facilitate telephone-related processes andfunctions; messaging instructions 862 to facilitate electronic-messagingrelated processes and functions; web browser instructions 864 tofacilitate web browsing-related processes and functions; mediaprocessing instructions 866 to facilitate media processing-relatedprocesses and functions; location services instructions including GPSand/or navigation instructions 868 and Wi-Fi based location instructionsto facilitate location based functionality; camera instructions 870 tofacilitate camera-related processes and functions; and/or other softwareinstructions 872 to facilitate other processes and functions, e.g.,security processes and functions, and processes and functions related tothe systems. The memory 850 may also store other software instructionssuch as web video instructions to facilitate web video-related processesand functions; and/or web shopping instructions to facilitate webshopping-related processes and functions. In some implementations, themedia processing instructions 866 are divided into audio processinginstructions and video processing instructions to facilitate audioprocessing-related processes and functions and video processing-relatedprocesses and functions, respectively. A mobile equipment identifier,such as an International Mobile Equipment Identity (IMEI) 874 or asimilar hardware identifier can also be stored in memory 850.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. The memory 850 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions may be implemented in hardware and/or in software, includingin one or more signal processing and/or application specific integratedcircuits.

FIG. 9 is a block diagram of one embodiment of a computing system 900.The computing system illustrated in FIG. 9 is intended to represent arange of computing systems (either wired or wireless) including, forexample, desktop computer systems, laptop computer systems, tabletcomputer systems, cellular telephones, personal digital assistants(PDAs) including cellular-enabled PDAs, set top boxes, entertainmentsystems or other consumer electronic devices, smart appliance devices,or one or more implementations of a smart media playback device.Alternative computing systems may include more, fewer and/or differentcomponents. The computing system of FIG. 9 may be used to provide thecomputing device and/or a server device to which the computing devicemay connect.

The computing system 900 includes bus 935 or other communication deviceto communicate information, and processor(s) 910 coupled to bus 935 thatmay process information. While the computing system 900 is illustratedwith a single processor, the computing system 900 may include multipleprocessors and/or co-processors. The computing system 900 further mayinclude random access memory 920 (RAM) or other dynamic storage devicecoupled to the bus 935. The memory 920 may store information andinstructions that may be executed by processor(s) 910. The memory 920may also be used to store temporary variables or other intermediateinformation during execution of instructions by the processor(s) 910.

The computing system 900 may also include read only memory (ROM) 930and/or another data storage device 940 coupled to the bus 935 that maystore information and instructions for the processor(s) 910. The datastorage device 940 can be or include a variety of storage devices, suchas a flash memory device, a magnetic disk, or an optical disc and may becoupled to computing system 900 via the bus 935 or via a remoteperipheral interface.

The computing system 900 may also be coupled, via the bus 935, to adisplay device 950 to display information to a user. The computingsystem 900 can also include an alphanumeric input device 960, includingalphanumeric and other keys, which may be coupled to bus 935 tocommunicate information and command selections to processor(s) 910.Another type of user input device includes a cursor control 970 device,such as a touchpad, a mouse, a trackball, or cursor direction keys tocommunicate direction information and command selections to processor(s)910 and to control cursor movement on the display device 950. Thecomputing system 900 may also receive user input from a remote devicethat is communicatively coupled via one or more network interface(s)980.

The computing system 900 further may include one or more networkinterface(s) 980 to provide access to a network, such as a local areanetwork. The network interface(s) 980 may include, for example, awireless network interface having antenna 985, which may represent oneor more antenna(e). The computing system 900 can include multiplewireless network interfaces such as a combination of WiFi, Bluetooth®,near field communication (NFC), and/or cellular telephony interfaces.The network interface(s) 980 may also include, for example, a wirednetwork interface to communicate with remote devices via network cable987, which may be, for example, an Ethernet cable, a coaxial cable, afiber optic cable, a serial cable, or a parallel cable.

In one embodiment the network interface(s) 980 may provide access to alocal area network, for example, by conforming to IEEE 802.11 b and/orIEEE 802.11 g standards, and/or the wireless network interface mayprovide access to a personal area network, for example, by conforming toBluetooth standards. Other wireless network interfaces and/or protocolscan also be supported. In addition to, or instead of, communication viawireless LAN standards, network interface(s) 980 may provide wirelesscommunications using, for example, Time Division, Multiple Access (TDMA)protocols, Global System for Mobile Communications (GSM) protocols, CodeDivision, Multiple Access (CDMA) protocols, Long Term Evolution (LTE)protocols, and/or any other type of wireless communications protocol.

The computing system 900 can further include one or more energy sources905 and one or more energy measurement systems 945. Energy sources 905can include an AC/DC adapter coupled to an external power source, one ormore batteries, one or more charge storage devices, a USB charger, orother energy source. Energy measurement systems include at least onevoltage or amperage measuring device that can measure energy consumed bythe computing system 900 during a predetermined period of time.Additionally, one or more energy measurement systems can be includedthat measure, e.g., energy consumed by a display device, coolingsubsystem, Wi-Fi subsystem, or other frequently used or high-energyconsumption subsystem.

FIG. 10 illustrates a block diagram of a virtual assistant system 1000,according to embodiments described herein. The illustrated virtualassistant system 1000 is exemplary of one embodiment and is not limitingas to all embodiments described herein. Virtual assistants employed bythe various embodiment described herein may include additional, fewerand/or different components or features than those illustrated. Thevirtual assistant system 1000 includes a virtual assistant 1002 that canaccept user input 1004, such as spoken or typed language, processes theinput, and generate output 1008 to the user and/or perform 1010 actionson behalf of the user. The virtual assistant 1002 can use contextinformation to supplement natural language or gestural input from auser. Context information can be used to clarify the intent of the userand to reduce the number of candidate interpretations of the user'sinput. The context information can also reduce the need for the user toprovide excessive clarification input. Context can include any availableinformation that is usable by the assistant to supplement explicit userinput to constrain an information-processing problem and/or topersonalize results. Context can be used to constrain solutions duringvarious phases of processing, including, for example, speechrecognition, natural language processing, task flow processing, anddialog generation.

The virtual assistant 1002 can draw on any of a number of differentbackground sources of knowledge and data, such as dictionaries, domainmodels, and/or task models. From the perspective of the presentlydescribed embodiments, such background sources may be internal to thevirtual assistant 1002 or can be gathered from one or more remotedatabases. In addition to user input 1004 and background sources, thevirtual assistant 1002 can also draw on information from several sourcesof context, including, for example, device sensor data 1056, applicationpreferences and usage history 1072, dialog history and assistant memory1052, personal databases 1058, personal acoustic context data 1080,current application context 1060, and event context 1006.

In one embodiment, a physical device running the virtual assistant 1002,such as a user device, playback device, or smart media playback deviceas described herein, have one or more sensors devices. Such sensors canprovide sources of contextual information in the form of device sensordata 1056. Examples of sensor information include, without limitation,the user's current location; the local time at the user's currentlocation; the position, orientation, and motion of the device on whichthe user is interacting; the current light level, temperature and otherenvironmental measures; the properties of the microphones and cameras inuse; the current networks being used, and signatures of connectednetworks, including Ethernet, Wi-Fi and Bluetooth. Signatures includeMAC addresses of network access points, IP addresses assigned, deviceidentifiers such as Bluetooth names, frequency channels and otherproperties of wireless networks. Sensors can be of any type includingfor example: an accelerometer, compass, GPS unit, altitude detector,light sensor, thermometer, barometer, clock, network interface, batterytest circuitry, and the like.

The current application context 1060 refers to the application state orsimilar software state that is relevant to the current activity of theuser. For example, the user could be using a text messaging applicationto chat with a particular person. The Virtual assistant 1002 need not bespecific to or part of the user interface of the text messagingapplication. Instead, the virtual assistant 1002 can receive contextfrom any number of applications, with each application contributing itscontext to inform the virtual assistant 1002. If the user is currentlyusing an application when the virtual assistant 1002 is invoked, thestate of that application can provide useful context information. Forexample, if virtual assistant 1002 is invoked from within an emailapplication, context information may include sender information,recipient information, date and/or time sent, subject, data extractedfrom email content, mailbox or folder name, and the like.

In one embodiment, information describing the user's applicationpreferences and usage history 1072 includes preferences and settings forvarious applications, as well usage history associated with thoseapplications. Application preferences and usage history 1072 is used ascontext for interpreting and/or operationalizing the user's intent orother functions of the virtual assistant 1002. Examples of suchapplication preferences and usage history 1072 include, withoutlimitation, shortcuts, favorites, bookmarks, friends lists, or any othercollections of user data about people, companies, addresses, phonenumbers, places, web sites, email messages, or any other references;recent calls made on the device; recent text message conversations,including the parties to the conversations; recent requests for maps ordirections; recent web searches and URLs; stocks listed in a stockapplication; recent songs or video or other media played; the names ofalarms set on alerting applications; the names of applications or otherdigital objects on the device; and the user's preferred language or thelanguage in use at the user's location.

Another source of context data is the personal database 1058 of a useron a device such as a phone, such as for example an address bookcontaining names and phone numbers. In one embodiment, personalinformation of the user obtained from personal databases 1058 are usedas context for interpreting and/or operationalizing the user's intent orother functions of the virtual assistant 1002. For example, data in auser's contact database can be used to reduce ambiguity in interpretinga user's command when the user referred to someone by first name only.Examples of context information that can be obtained from personaldatabases 1058 include, without limitation, the user's contact database(address book)—including information about names, phone numbers,physical addresses, network addresses, account identifiers, importantdates—about people, companies, organizations, places, web sites, andother entities that the user might refer to; the user's own names,preferred pronunciations, addresses, phone numbers, and the like; theuser's named relationships, such as mother, father, sister, boss, andthe like; the user's calendar data, including calendar events, names ofspecial days, or any other named entries that the user might refer to;the user's reminders or task list, including lists of things to do,remember, or get that the user might refer to; names of songs, genres,playlists, and other data associated with the user's music library thatthe user might refer to; people, places, categories, tags, labels, orother symbolic names on photos or videos or other media in the user'smedia library; titles, authors, genres, or other symbolic names in booksor other literature in the user's personal library.

Another source of context data is the user's dialog history with thevirtual assistant 1002. Such history may include, for example,references to domains, people, places, and so forth. For example, a usercan ask “What's the time in New York?”. The virtual assistant 1002 canrespond by providing the current time in New York City. The user canthen ask “What's the weather?”. The virtual assistant 1002 ca use theprevious dialog history to infer that the location intended for theweather query is the last location mentioned in the dialog history.

Examples of context information from dialog history and virtualassistant memory include, without limitation, people mentioned in adialog; places and locations mentioned in a dialog; current time framein focus; current application domain in focus, such as email orcalendar; current task in focus, such as reading an email or creating acalendar entry; current domain objects in focus, such as an emailmessage that was just read or calendar entry that was just created;current state of a dialog or transactional flow, such as whether aquestion is being asked and what possible answers are expected; historyof user requests; history of results of user requests, such as sets ofrestaurants returned; history of phrases used by the assistant indialog; and facts that were told to the assistant by the user.

In one embodiment, personal acoustic context data 1080 be used to selectfrom possible statistical language models that may be used to understanduser speech, or otherwise tune the speech recognition to optimize forrecognized acoustical contexts. When interpreting speech input, thevirtual assistant 1002 can tune a speech to text service to take intoaccount the acoustic environments in which the speech is entered. Forexample, the noise profiles of a quiet office are different from thoseof automobiles or public places. If a speech recognition system canidentify and store acoustic profile data, these data can also beprovided as contextual information. When combined with other contextualinformation such as the properties of the microphones in use, thecurrent location, and the current dialog state, acoustic context can aidin recognition and interpretation of input.

In the foregoing specification, the invention has been describedregarding specific embodiments thereof. It will, however, be evidentthat various modifications and changes can be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. The specifics in thedescriptions and examples provided may be used anywhere in one or moreembodiments. The various features of the different embodiments orexamples may be variously combined with some features included andothers excluded to suit a variety of different applications. Examplesmay include subject matter such as a method, means for performing actsof the method, at least one machine-readable medium includinginstructions that, when performed by a machine cause the machine toperforms acts of the method, or of an apparatus or system according toembodiments and examples described herein. Additionally variouscomponents described herein can be a means for performing the operationsor functions described in accordance with an embodiment.

Embodiments described herein provide a media playback system thatenables a smooth transition between incoming audio events, such aspre-set alarms, and currently playing media. One embodiment provides fora media playback device comprising a memory device to storeinstructions, one or more processors to execute the instructions storedon the memory device, the instructions to cause the one or moreprocessors to provide a playback queue manager to manage one or moremedia playback queues including a set of media items associated with ascheduled event; and a playback routing manager to determine an outputdestination for the media items based on context associated with thescheduled event.

In one embodiment the playback queue manager is configured to managemultiple simultaneous playback queues, the multiple simultaneousplayback queues including one or more past, present or future mediaelements to be played. The playback manager can route output of playbackof the set of media items to one or more of multiple different connectedmedia playback devices based on the context associated with thescheduled event. In one embodiment the playback routing manageradditionally configurable to route the set of media items to one or moreof multiple different connected media playback devices for playback, theplayback routing manager to route the set of media items based on thecontext associated with the scheduled event. The context associated withthe scheduled event can include a user account associated with thescheduled event and can additionally include a location associated withthe media playback device. In one embodiment, the location associatedwith the media playback device includes a room or a location within theroom.

In one embodiment, the set of media items includes an audio alarm. Whenthe scheduled event is an alarm event the playback routing manager canto route the audio alarm to one or more of multiple different mediaplayback devices based on the context associated with the alarm event.In one embodiment, the context associated with the scheduled eventincludes a label associated with the scheduled event. The outputdestination for the set of media items can speak the label associatedwith the scheduled event or display the label associated with thescheduled event.

One embodiment provides for a non-transitory machine readable mediumstoring instructions which, when executed by one or more processors ofan electronic device, cause the one or more processors to performoperations including receiving a request to play a queue of media itemassociated with a scheduled event; determining whether an existingplayback queue is in effect on the electronic device; determiningwhether the queue of media items associated with the scheduled event isa transient queue; storing the existing playback queue as a secondarymedia queue in response to determining that the existing playback queueis in effect and that the queue of media items associated with thescheduled event is a transient queue; and playing the queue of mediaitems associated with the scheduled event until the scheduled event isdismissed.

In one embodiment, the operations additionally include determiningwhether a playback queue is in effect within a zone associated with theelectronic device and loading the queue of media items into the playbackqueue in response to determining that the playback queue is notcurrently effect within the zone. In one embodiment the instructions tocause additional operations including determining that an existingplayback queue is in effect within the zone associated with theelectronic device; temporarily detaching the electronic device from thezone; loading the queue of media items into the playback queue of theelectronic device; initiating playback of the queue of media items onthe electronic device; and restoring the electronic device to the zoneafter playback of the queue of media items.

One embodiment provides for a data processing system comprising a memorydevice to store instructions; one or more processors to execute theinstructions stored on the memory device, the instructions to cause theone or more processors to provide a virtual assistant to receive voiceinput at a media playback device associated with the data processingsystem, the virtual assistant, via the one or more processors, toreceive a voice command at the media playback device, the voice commandto schedule an event; determine a user account associated with the voicecommand; determine a type of command based on context informationassociated with the voice command; and determine a target device towhich media playback associated with the event is to be scheduled.

In one embodiment the virtual assistant, via the one or more processors,can output a verbal request for confirmation of a determined type ofcommand and target device. The virtual assistant, via the one or moreprocessors, can also determine an alternate type of command or targetdevice based on a correction received in response to the verbal requestfor confirmation. Furthermore, the virtual assistant can schedule theevent and associated media playback to the target device.

Other features of the present embodiments will be apparent from theaccompanying drawings and from the detailed description above.Accordingly, the true scope of the embodiments will become apparent tothe skilled practitioner upon a study of the drawings, specification,and following claims.

What is claimed is:
 1. A data processing system comprising: a memorydevice to store instructions; one or more processors to execute theinstructions stored on the memory device, the instructions to cause theone or more processors to provide a virtual assistant to receive voiceinput at a media playback device associated with the data processingsystem, the virtual assistant, via the one or more processors, to:receive a voice command at the media playback device, the voice commandto schedule an event; determine context information associated with thevoice command, the context information including a user accountassociated with the voice command, wherein the user account is a useraccount in a family of related user accounts and the user account isassociated with the media playback device; determine a type of commandbased on context information associated with the voice command;determine a target device to which media playback associated with theevent is to be scheduled; and schedule the event to a determined targetdevice based on a determined type of command.
 2. The data processingsystem as in claim 1, the virtual assistant, via the one or moreprocessors, to output a verbal request for confirmation of the type ofcommand and target device determined by the virtual assistant.
 3. Thedata processing system as in claim 2, the virtual assistant, via the oneor more processors, to determine an alternate type of command oralternate target device based on a correction received in response tothe verbal request for confirmation.
 4. The data processing system as inclaim 3, wherein to schedule the event to the determined target devicebased on the determined type of command includes to schedule the eventto the alternate target device based on the alternate type of command.5. The data processing system as in claim 1, wherein the contextinformation associated with the voice command includes a locationassociated with the media playback device.
 6. The data processing systemas in claim 5, wherein the location associated with the media playbackdevice includes a room associated with the media playback device.
 7. Thedata processing system as in claim 6, wherein the location associatedwith the media playback device additionally includes a location withinthe room.
 8. A non-transitory machine readable medium storinginstructions which, when executed by one or more processors of a dataprocessing system, cause the data processing system to performoperations comprising: receiving a voice command to schedule an event,the voice command received by a virtual assistant at a media playbackdevice associated with the data processing system; determining contextinformation associated with the voice command, the context informationincluding a user account associated with the voice command, wherein theuser account is a user account in a family of related user accounts andthe user account is associated with the media playback device;determining a type of command based on context information associatedwith the voice command; determining a target device to which mediaplayback associated with the event is to be scheduled; and schedulingthe event to the determined target device based on a determined type ofcommand.
 9. The non-transitory machine readable medium as in claim 8,the operations further comprising outputting, via the virtual assistant,a verbal request for confirmation of the type of command and the targetdevice determined by the virtual assistant.
 10. The non-transitorymachine readable medium as in claim 9, the operations further comprisingdetermining, via the virtual assistant, an alternate type of command oralternate target device based on a correction received in response tothe verbal request for confirmation.
 11. The non-transitory machinereadable medium as in claim 10, wherein to schedule the event to thedetermined target device based on the determined type of commandincludes to schedule the event to the alternate target device based onthe alternate type of command.
 12. The non-transitory machine readablemedium as in claim 8, wherein the context information associated withthe voice command includes a location associated with the media playbackdevice.
 13. The non-transitory machine readable medium as in claim 12,wherein the location associated with the media playback device includesa room associated with the media playback device.
 14. The non-transitorymachine readable medium as in claim 13, wherein the location associatedwith the media playback device additionally includes a location withinthe room.
 15. A media playback device comprising: a memory device tostore instructions; one or more processors to execute the instructionsstored on the memory device, the instructions to cause the one or moreprocessors to provide: a virtual assistant to receive voice input at themedia playback device, the virtual assistant, via the one or moreprocessors, to: receive a voice command at the media playback device,the voice command to schedule an event; determine context informationassociated with the voice command, the context information including auser account associated with the voice command, wherein the user accountis a user account in a family of related user accounts and the useraccount is associated with the media playback device; determine a typeof command based on context information associated with the voicecommand; determine a target device to which media playback associatedwith the event is to be scheduled; and schedule the event to adetermined target device based on a determined type of command.
 16. Themedia playback device as in claim 15, the virtual assistant, via the oneor more processors, to output a verbal request for confirmation of thetype of command and target device determined by the virtual assistant.17. The media playback device as in claim 16, the virtual assistant, viathe one or more processors, to determine an alternate type of command oralternate target device based on a correction received in response tothe verbal request for confirmation.
 18. The media playback device as inclaim 17, wherein to schedule the event to the determined target devicebased on the determined type of command includes to schedule the eventto the alternate target device based on the alternate type of command.19. The media playback device as in claim 15, wherein the contextinformation associated with the voice command includes a locationassociated with the media playback device.
 20. The media playback deviceas in claim 19, wherein the location associated with the media playbackdevice includes a room associated with the media playback device or alocation within the room associated with the media playback device.